Render EMF/WMF diagrams so Office figures are analyzed - #1279
Merged
Paul Lizer (paullizer) merged 5 commits intoAug 18, 2026
Merged
Conversation
Refs #1277 Images embedded in Word and PowerPoint as EMF or WMF metafiles were silently skipped, because the extraction whitelist only accepted raster formats. Word stores pasted diagrams, SmartArt, Visio drawings, and charts this way, so architecture diagrams -- often the densest figures in a document -- were never analyzed or indexed. A real TIC 3.0 document uploaded with Enhanced extraction on contained four EMF diagrams and produced zero image analysis and zero log output. Rasterizing them is not straightforward here. Pillow only installs a metafile renderer on Windows, where it is backed by GDI, and the application container is Linux distroless -- no shell and no package manager -- so an external converter such as LibreOffice or Inkscape is not an option either. functions_emf_render.py therefore renders metafiles in-process using only Pillow, io, and struct, so behavior is identical on every platform and the container is unchanged. It covers the record subset Office emits for diagrams: path construction, filled and stroked polygons, Bezier curves, rectangles and ellipses, pen and brush objects, world transforms, and text runs. Unsupported records are skipped rather than failing the render, so output degrades in fidelity instead of disappearing. Text drawn inside a metafile is extracted as well, so figure labels stay searchable even when the vision engine returns nothing. Verified against the four real diagrams: all render with plausible ink coverage and recover their labels, including Event Hub Namespace, Log Analytics Workspace, and Azure Firewall Table. Embedded image processing is also no longer silent. A document whose images were all skipped was indistinguishable from one with no images, which is what made the original problem impossible to diagnose from the workspace. Extraction now reports candidates found, images analyzed, and per-reason skip counts; progress is reported per image; and the counts are persisted on the document. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
11 tasks
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…ith 'import' and 'import from'' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1277
Problem
A real TIC 3.0 architecture document uploaded with Enhanced extraction turned on produced no image analysis and no log output at all, despite containing four diagrams.
Root cause: the images were EMF metafiles, and the embedded-image whitelist only accepted raster formats.
Word stores pasted diagrams, SmartArt, Visio drawings, and charts as EMF, so this silently skipped what are often the most information-dense figures in a document. Worse, extraction returned before emitting any status message, so "no images in this file" and "four images found, all skipped" looked identical in the workspace log — which is what made the problem impossible to diagnose.
Why this isn't a one-line whitelist change
Neither Document Intelligence nor Content Understanding accepts a metafile, so EMF has to be rasterized first, and the usual options are all unavailable here:
hasattr(Image.core, "drawwmf"), commented "windows only", because it is GDI-backed. Relying on it would work in local Windows development and silently fail in the container.mcr.microsoft.com/azurelinux/distroless/python:3.12— no shell, no package manager — so LibreOffice or Inkscape would mean copying a very large dependency tree into a distroless image.Approach
functions_emf_render.pyrenders EMF and WMF in-process using onlyPIL,io, andstruct. No system packages, no external binaries, no platform branches, so behavior is identical in the container and locally. The container is unchanged.It covers the record subset Office actually emits for diagrams: path construction, filled and stroked polygons, Bezier curves, rectangles and ellipses, pen and brush objects, world transforms, and text runs. Records outside that subset are skipped rather than failing the render, so output degrades in fidelity instead of disappearing.
Text drawn inside a metafile is extracted as well and attached to the chunk, so figure labels stay searchable even when the vision engine returns no description.
Verified against the real document
All four diagrams now rasterize, and I inspected the rendered output rather than trusting an exit code — they are recognizable Azure architecture diagrams with subscription boundaries, service icons, and connectors. Recovered labels include Event Hub Namespace, Log Analytics Workspace, Azure Firewall Table, SysLog Table, and CLAW Registered App.
Two bugs surfaced during that verification and are fixed here: the EMF/WMF format sniff misfired because
\x01\x00is just the first half of the EMF record type (the reliable discriminator is the" EMF"signature at offset 40), and theEXTTEXTOUTWfield offsets were wrong, which silently yielded zero text runs.Observability
Embedded image processing is no longer silent. Extraction now reports how many image parts were found, how many were analyzed, and why any were skipped, with progress reported per image:
The counts persist on the document as
office_embedded_image_candidates,office_embedded_image_count, andoffice_embedded_image_skipped, so the outcome can be confirmed after processing finishes.Testing
Six new tests covering metafile rasterization, format-detection failure reporting, and diagnostics that distinguish "no images" from "images found but all skipped" — 17/17 in the embedded image suite. The Content Understanding, legacy Word, Document Intelligence, and route policy suites all still pass.
Reviewer notes
Version bumped to
0.250.223.